build(salesforcedx-vscode): update node to v16 and switch from npm to yarn #4042

mohanraj-r · 2022-04-19T18:13:12Z

What does this PR do?

Update node to Active LTS v16 (EOL 2024-04-30)
Switch from npm to yarn due to issues with npm 8 and vsce ~~Update to npm v8 which is bundled with Node v16~~
~~Rebased on build: add typescript project references #4055~~

What issues does this PR fix or reference?

@W-11004197@

Functionality Before

Node v14 was used with older lockfile format

Functionality After

Node v16 used with yarn ~~npm v8 and lockfile format v3~~

Notes

Run npm run reinstall after merge (remind team in slack)

Summary

vsce packaging doesn't work with
- npm v8 / v7 (even on node 14) (Support npm workspaces microsoft/vscode-vsce#580, Add --ignoreDependencies to package command microsoft/vscode-vsce#439, Support relative paths in depedencies microsoft/vscode-vsce#673, Peer/Dev Dependency Resolution Error lerna/lerna#2822)
- yarn classic / berry (with legacy node_modules)
  - vsce packaging doesn't error but doesn't produce valid working extensions (Created VSIX via vsce package --yarn does not include not-hoisted deep dependency microsoft/vscode-vsce#432)
    - 114 MB vs 476 MB with npm

Solutions

use our own bundling ([WIP] build: use a bundler (esbuild) along with vsce to package vsix files #4087)
- esbuild
  - doesn't support dynamic imports
- rollup
  - has a plugin for dynamic imports that I am trying out right now
Downgrade to using npm v6 with node v16 (build: update to node v16 with npm v6 for packaging #4092)

PNPM

Pros

Built-in phantom dep, peer dep checks
Built-in support for mono-repo
- Handles dep locks for all packages in the mono-repo
Almost drop-in replacement for npm commands
- supports node_modules by default unlike latest yarn
Closest drop-in replacement for lerna with similar commands (run, exec) and options (concurrency, filtering/scope)
- bootstraps the mono-repo packages
Easier to read output / mental model
Lock file doesn't get updated implicitly on each install
Faster than npm, yarn

TODO

Packaging vsix fix

Build fix

Refactor

Document / share

Explore

yarn install --flat
- Install all the dependencies, but only allow one version for each package.
Yarn v3 with node_modules
yarn install --frozen-lockfile
- Don’t generate a yarn.lock lockfile and fail if an update is needed.
yarn audit | Yarn
yarn upgrade-interactive

gbockus-sf

Cool looks good 👍 Any ideas on why the circle build is failing? The error looks familiar but I don't remember the details of what results in that error.

npm WARN config production Use `--omit=dev` instead.
 ERROR  Command failed: npm list --production --parseable --depth=99999 --loglevel=error

Think we need to update the packing script to account for the above warning?

mohanraj-r · 2022-04-26T19:53:50Z

vsce package doesn't work with npm 8 or 7 (microsoft/vscode-vsce#580, microsoft/vscode-vsce#439, microsoft/vscode-vsce#673, lerna/lerna#2822) even on node 14.
Also tried

with and without npm workspaces
updating vsce to latest version (2.x).

lerna ERR! npm run vscode:package stderr:
npm WARN config production Use `--omit=dev` instead.
 ERROR  Command failed: npm list --production --parseable --depth=99999 --loglevel=error
npm ERR! code ELSPROBLEMS
npm ERR! missing: @salesforce/[email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/mocha@^5, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/rimraf@^2.0.2, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @types/sinon@^2.3.7, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: chai@^4.0.2, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: mocha-junit-reporter@^1.23.3, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: mocha-multi-reporters@^1.1.7, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: mocha@^5, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: nyc@^13, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: sinon@^13.0.1, required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-apex-debugger@npm:@salesforce/[email protected]
npm ERR! missing: @salesforce/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/mocha@^5, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/sinon@^2.3.7, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: @types/[email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: chai@^4.0.2, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: mocha-junit-reporter@^1.23.3, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: mocha-multi-reporters@^1.1.7, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: mocha@^5, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: nyc@^13, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: sinon@^13.0.1, required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]
npm ERR! missing: [email protected], required by salesforcedx-utils-vscode@npm:@salesforce/[email protected]

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2022-04-19T18_36_10_092Z-debug-0.log

mohanraj-r · 2022-04-27T20:19:14Z

Bundle size / validity

Baseline (vsce + npm prune)

To compare with baseline, the vsix files buit with vsce & npm (from develop@ 6782d42)

 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-debugger/salesforcedx-vscode-apex-debugger-54.11.0.vsix (605 files, 1.24MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.11.0.vsix (4830 files, 6.93MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex/salesforcedx-vscode-apex-54.11.0.vsix (5193 files, 22.06MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.11.0.vsix (16070 files, 24.6MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.11.0.vsix (6 files, 12.04KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.11.0.vsix (5524 files, 8.6MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.11.0.vsix (20187 files, 30.9MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.11.0.vsix (3231 files, 4.84MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.11.0.vsix (757 files, 3.22MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.11.0.vsix (7 files, 21.57KB)

Yarn

vsce packaging seems to work with yarn (without any patches) - but the vsix bundle sizes are significantly lower than those produced with npm. Quick smoke test with the vsix files shows errors and extensions not working as intended.

8:scode-apex-debugger/salesforcedx-vscode-apex-debugger-54.10.0.vsix (28 files, 48.31KB)
13:scode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.10.0.vsix (52 file
18:scode-apex/salesforcedx-vscode-apex-54.10.0.vsix (109 files, 14.85MB)
22: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.10.0.vsix (295 files, 373.02KB)
25: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.10.0.vsix (6 files, 12.03KB)
28: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.10.0.vsix (19 files, 55.78KB)
31: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.10.0.vsix (131 files, 166.16KB)
34: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.10.0.vsix (9 files, 402.21KB)
37: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.10.0.vsix (22 files, 45.4KB)
40: DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.10.0.vsix (7 files, 21.48KB)

With esbuild


 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-debugger/salesforcedx-vscode-apex-debugger-54.11.0.vsix (28 files, 61.62KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex-replay-debugger/salesforcedx-vscode-apex-replay-debugger-54.11.0.vsix (52 files, 1.23MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-apex/salesforcedx-vscode-apex-54.11.0.vsix (109 files, 16.15MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-core/salesforcedx-vscode-core-54.11.0.vsix (295 files, 3.69MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-expanded/salesforcedx-vscode-expanded-54.11.0.vsix (6 files, 12.05KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lightning/salesforcedx-vscode-lightning-54.11.0.vsix (19 files, 801.67KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-lwc/salesforcedx-vscode-lwc-54.11.0.vsix (131 files, 1.22MB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-soql/salesforcedx-vscode-soql-54.11.0.vsix (9 files, 391.29KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode-visualforce/salesforcedx-vscode-visualforce-54.11.0.vsix (22 files, 73.99KB)
 DONE  Packaged: salesforcedx-vscode/packages/salesforcedx-vscode/salesforcedx-vscode-54.11.0.vsix (7 files, 21.58KB)

Update vsce

Using latest vsce version 2.7.0 (or 2.6.1) results in error from package that doesn't need to be packaged - microsoft/vscode-vsce#675

$ lerna run vscode:package --concurrency 1 && node scripts/reformat-with-prettier
lerna notice cli v3.22.1
lerna info Executing command in 10 packages: "yarn run vscode:package"
lerna ERR! yarn run vscode:package exited 1 in 'salesforcedx-vscode-apex-debugger'
lerna ERR! yarn run vscode:package stdout:
$ vsce package --yarn
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

lerna ERR! yarn run vscode:package stderr:
 ERROR  Extension entrypoint(s) missing. Make sure these files exist and aren't ignored by '.vscodeignore':
  extension/out/src.js
error Command failed with exit code 1.

lerna ERR! yarn run vscode:package exited 1 in 'salesforcedx-vscode-apex-debugger'
lerna WARN complete Waiting for 1 child process to exit. CTRL-C to exit immediately.
error Command failed with exit code 1.

Other potential solutions

npm prune equivalent behavior · Issue #696 · yarnpkg/yarn doesn't help
upgrading yarn, vsce, lerna or using vsce-yarn-patch individually or in combinations doesn't help

…t v3

as it has issues with the tsconfig used by tsc build with project ref

…t v3

This reverts commit 041242d.

refactor clean script

…med without a reference

microsoft/TypeScript#42873

mohanraj-r · 2022-05-09T21:57:25Z

Broken up into smaller PRs

mohanraj-r requested a review from a team as a code owner April 19, 2022 18:13

mohanraj-r requested a review from gbockus-sf April 19, 2022 18:13

gbockus-sf reviewed Apr 21, 2022

View reviewed changes

mohanraj-r changed the title ~~build(salesforcedx-vscode): update node to v16 and npm to v8~~ build(salesforcedx-vscode): update node to v16 and switch from npm to yarn Apr 28, 2022

mohanraj-r mentioned this pull request Apr 28, 2022

build(salesforcedx-vscode): update node to v16 and npm v8 #4067

Closed

mohanraj-r added 23 commits May 3, 2022 14:02

ci: update CI configs, dev docs to node v16

1f5d682

build: update node to v16, npm to v8 and switch to new lockfile forma…

ba6cc20

…t v3

ci: update circle node version to one that exists

966ddd6

build: downgrade node to the version available in CircleCI

49cf42e

test: add check for typescript proj refs

791ae3f

build: add compile scripts, fix tsconfig in packages

73d2412

refactor: common tsconfig to reuse node 14 base config

5b8aea7

refactor: remove tsc clean from existing clean script

2d27ce5

fix: add separate tsconfig for ts-loader / webpack

a9f824d

as it has issues with the tsconfig used by tsc build with project ref

fix(soql): add tsconfig for webpack / ts-loader

069e569

build: revert to es6 target in tsconfig

ddd2329

refactor: remove unnecessary tsconfig webpack config files

1490052

build: update node to v16, npm to v8 and switch to new lockfile forma…

c5becb7

…t v3

build: update node, npm version in lock file

d43fce5

build: enable npm workspaces (WIP)

177da33

build: replace npm lock file with yarn lock

9fe07d4

chore: update after rebase onto typescript proj ref enable

32e70e9

fix(compile): silence compile errors with ts-ignore

e5cb052

fix(compile): remove extraneous node_modules from import paths

e9fb07a

refactor: switch to vsce packaging with yarn; remove npm prune

07b0d68

chore: lint fix alphabetically sorted imports

31f5d1b

ci: add yarn and replace npm ci with yarn install frozen lock file equiv

ffe49ce

ci: update yarn engine version to match ci env

5ec68bb

Mohan Raj Rajamanickam added 16 commits May 3, 2022 14:12

Revert "build: switch to yarn 3 with legacy node modules"

26f1c54

This reverts commit 041242d.

build: switch to esbuild for prepublish

677b370

build: switch to latest version of vsce

1783d0d

build: disable dependency detection via npm or yarn for vsce package

d8b4510

build: switch to pnpm

795c12a

refactor: remove obsolete lerna bootstrap

3b93bda

refactor: remove obsolete check peer deps

7ca6581

refactor: esbuild run script to include minify option

0b725f7

build: add missing deps, peer deps

7da9913

build: replace npm in engine and deps with pnpm

a9905e2

build: remove vscode ignore files to debug esbuild

51573e3

build: enable debug logging for esbuild

81c53d8

build: add script to setup pnpm and node

89c1910

refactor clean script

refactor: compile and packaging scripts to use pnpm

8a590f3

build: fix @types/sinon error - the inferred type of "X" cannot be na…

eb7268d

…med without a reference

chore: post rebase fix merge conflicts

d96e324

mohanraj-r force-pushed the mohanraj-r/node16_npm8_update branch from b892421 to d96e324 Compare May 3, 2022 21:17

Mohan Raj Rajamanickam added 9 commits May 3, 2022 14:24

chore: remove obsolete npm package lock, regen pnpm lock (post rebase)

f1fc823

build: fix webpack error with pnpm

9305dd3

microsoft/TypeScript#42873

build: fix synthetic default imports in core pkg

5214837

build: use rollup to bundle core pkg

a5de013

refactor: move rollup config, deps to root

b9d82ea

build: switch from esbuild to using rollup for all pkgs

407219b

build: fix synthetic imports, missing license, normalize entry point

194992a

build: add dynamic import rollup plugin

218174d

build: add script to install built vsix files to vscode insider

7d77584

mohanraj-r mentioned this pull request May 9, 2022

[WIP] build: use a bundler (esbuild) along with vsce to package vsix files #4087

Closed

mohanraj-r closed this May 9, 2022

mohanraj-r mentioned this pull request May 10, 2022

build: update to node v16 with npm v6 for packaging #4092

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build(salesforcedx-vscode): update node to v16 and switch from npm to yarn #4042

build(salesforcedx-vscode): update node to v16 and switch from npm to yarn #4042

mohanraj-r commented Apr 19, 2022 •

edited

Loading

gbockus-sf left a comment

mohanraj-r commented Apr 26, 2022 •

edited

Loading

mohanraj-r commented Apr 27, 2022 •

edited

Loading

mohanraj-r commented May 9, 2022

build(salesforcedx-vscode): update node to v16 and switch from npm to yarn #4042

build(salesforcedx-vscode): update node to v16 and switch from npm to yarn #4042

Conversation

mohanraj-r commented Apr 19, 2022 • edited Loading

What does this PR do?

What issues does this PR fix or reference?

Functionality Before

Functionality After

Notes

Summary

Solutions

PNPM

TODO

Packaging vsix fix

Build fix

Refactor

Document / share

Explore

gbockus-sf left a comment

Choose a reason for hiding this comment

mohanraj-r commented Apr 26, 2022 • edited Loading

mohanraj-r commented Apr 27, 2022 • edited Loading

Bundle size / validity

Baseline (vsce + npm prune)

Yarn

With esbuild

Update vsce

Other potential solutions

mohanraj-r commented May 9, 2022

mohanraj-r commented Apr 19, 2022 •

edited

Loading

mohanraj-r commented Apr 26, 2022 •

edited

Loading

mohanraj-r commented Apr 27, 2022 •

edited

Loading